There are many applications for automatic classification of items such as documents, images, and recordings. To address this need, a plethora of classifiers have been developed. Examples include a priori rule-based classifiers, such as expert systems, and classifiers based on probabilistic dependency models learned from training data. Classifiers based on probabilistic dependency models include classifiers based on decision trees models, support vector machines, Bayesian belief networks, and neural networks.
Within each of these model types, varying the model assumption and/or the training technique can produce different classifiers. For example, different neural network models result depending on the number of levels selected and the number of nodes within each level. As a rule for complex classification problems, each classifier produces at least slightly different results and none of the classifiers provides correct classifications for every instance.
To improve classifier performance, it has been proposed to combine multiple classifiers to produce a single meta-classifier. One approach to combining classifiers has been to take a vote among the classifiers and classify items based on majority vote. Optionally, the votes can be weighted by the confidence levels the classifiers express with respect to their decisions. Another approach is to believe the classifier that expresses the highest confidence in its decision, i.e., provides the highest estimated probability that its decision is correct. Where classifiers do not provide probability estimates, probability estimates have been generated based on the classifier's performances on a fixed number of “nearest neighbors” in the training set. A further approach takes N classifiers j whose output is pj(ci|x), the probability that classification ci is correct given input x, and selects the classification ci that maximizes:       1    N    ⁢            ∑              j        =        1            N        ⁢                   ⁢                  p        j            ⁡              (                              c            i                    |          x                )            
While these meta-classifiers often work better than any of the individual classifiers they combine, these meta-classifiers still commonly make mistakes. Thus, there remains an unsatisfied need for a meta-classifier or other classifier that generally makes fewer or less costly mistakes than currently available classifiers.